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This  publication  Is  priaarlly  a  working  paper.  It  Is  published  solely  to  dociawnt  work  perforaed 


SUMURY 


This  paper  describes  the  results  of  a  review  of  Air  Force  logistics  capability  assessment 
modeling  undertaken  by  tbe  Logistics  and  Human  Factors  Division  of  the  Air  Force  Human  Resources 
Laboratory.  The  goal  of  this  effort  was  to  determine  wiys  to  apply  technology  advances  In 
computer  hardware  and  software  to  Improve  combat  operations  and  logistics  capability  evaluations. 
This  paper  concentrates  upon  the  Initial  findings  of  the  study  with  respect  to  current 
computer-based  models,  and  the  criteria  that  one  can  use  to  evaluate  these  models.  Multiple 
sources  were  used  to  collect  data.  Information  was  collected  through  a  combination  of  written 
surveys.  Interviews  with  users,  and  a  literature  review.  Users  of  models  were  Interviewed  at 
organizations  within  HQ  USAF,  major  commands,  various  other  military  agencies,  and  the  RAND 
Corporation.  The  paper  then  provides  a  set  of  shortfalls  Identified  by  users  regarding  current 
models'  capabilities.  Following  this,  a  set  of  model  evaluation  criteria  Is  proposed,  dealing 
with  such  features  as:  mission  effectiveness,  cost  effectiveness,  realism,  documentation,  and 
design  features.  Examples  are  used  to  Illustrate  a  model  evaluation  technique  and  to  Illustrate 
tradeoffs  between  the  proposed  criteria.  The  evaluation  method  can  be  used  to  help  Judge  the 
relative  merits  of  alternative  modeling  systems. 
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PREFACE 


This  paper  describes  the  results  of  a  review  of  Air  Force  logistics  capability 
assessment  modeling  undertaken  by  the  Logistics  and  Human  Factors  Division  of  the  Air 
Force  Human  Resources  Laboratory  (AFHRL).  This  study  Is  part  of  a  larger  effort  whose 
ultimate  goal  Is  to  develop  Improved  combat  operations  and  logistics  capability 

evaluations.  The  approach  Is  based  on  the  premise  that  technological  advances  In 
computer  hardware  and  software  could  be  applied  to  address  widespread  perceived 

deficiencies  In  current  logistics  capability  assessment  modeling  techniques.  The 
Initial  study  described  here  Involved  Investigating  the  views  of  current  model  users, 
consulting  with  subject-matter  experts,  and  reviewing  relevant  literature.  The  focus  of 
this  paper  Is  on  describing  findings  and  model  evaluation  techniques.  A  separate  paper. 
The  Productivity  Improvements  In  Simulation  Modeling  (PRISM)  Project:  Concepts  and 
Motivations  (Popken,  1988),  discusses  the  specific  technical  approach  AFHRL  Is  taking  to 
address  the  Identified  modeling  deficiencies. 

The  authors  would  like  to  acknowledge  the  contributions  of  Captains  Tom  King  and 

Maureen  Harrington.  These  Individuals  were  Instrumental  In  establishing  the 

requirements  for  and  Initial  direction  to  the  user  needs  study.  They  logged  many  miles 
to  ensure  that  the  project  received  this  vital  user  Input. 
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ANALYSIS  OF  AIR  FORCE  LOGISTICS 
CAPABILITY  ASSESSMENT  MODELS 


I.  INTRODUCTION 

In  1987,  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  Initiated  an  effort  to  determine 
the  simulation  modeling  needs  of  Air  Force  organizations  which  conduct  logistics  capability 
assessment.  The  major  focus  was  on  discrete-event  simulation  models  of  logistics  support 
systems.  Several  simulation  models  of  air  combat  were  also  Investigated  for  relevant  Information 
on  modeling  practices  and  needs.  In  addition.  Information  was  gathered  on  computer  hardware  and 
software  technologies  that  could  prove  useful  In  meeting  user-identified  needs.  AFHRL  was 
assisted  In  this  effort  by  Systems  Exploration  Incorporated  (SEI). 

The  findings  of  the  user  needs  study  Indicated  the  need  for  substantial  Improvement  In  the 
types  of  models  and  modeling  support  available  for  logistics  capability  assessment.  In  general, 
the  existing  models  were  found  to  be  difficult  to  use.  Insufficiently  documented,  and  difficult 
to  modify,  and  to  require  Inordinate  amounts  of  data  preparation. 

The  purpose  of  this  paper  Is  to  present  the  logistics  modeling  community  with  a  comprehensive 
discussion  of  the  advantages,  disadvantages,  and  tradeoffs  present  In  the  simulation  models 
currently  used  for  logistics  capability  assessment.  The  paper  begins  with  a  discussion  of  the 
data  collection  method.  User-Identified  model  deficiencies  are  then  provided.  The  paper  then 
defines  a  set  of  model  evaluation  criteria.  A  method  Is  described  for  using  the  criteria  to 
prioritize  model  selection  or  Improvements  In  terms  of  organizational  priorities,  resource 
requirements,  and  the  current  models'  capabilities.  The  Intent  Is  for  organizations  to  apply  or 
adapt  the  evaluation  methodology  to  their  own  situations.  Those  Intending  to  use  or  Improve 
logistics  capability  assessment  models  will  have  a  framework  by  which  to  Judge  the 
appropriateness  of  candidate  models. 


II.  DATA  COLLECTION 

Identification  of  modeling  deficiencies  was  accomplished  through  a  combination  of  written 
surveys.  Interviews  with  users,  and  literature  review.  To  begin,  an  Initial  survey  was  designed 
to  Identify  United  States  Air  Force  (USAF)  and  other  personnel  who  used  military  models  or  the 
output  of  military  models.  A  brief  survey  form  was  designed  and  a  mailing  list  was  compiled  from 
attendance  rosters  of  recent  meetings  and  conferences  on  the  subjects  of  capability  assessment, 
military  modeling,  and  wargaming.  This  survey  was  limited  to  the  following  questions: 

1.  Do  you  currently  use  simulation  models?  Which  ones? 

2.  Name,  address,  phone  number? 

3.  Planned  upgrades  or  changes? 

4.  Category  of  use?  (See  the  three  categories  given  below) 

This  survey  was  useful  In  Identifying  the  scope  of  modeling  efforts  and  In  establishing  points  of 
contact  for  the  Interviews. 

By  far  the  most  effective  of  the  data  collection  approaches  was  the  Interview  process.  Those 
Interviewed  can  be  grouped  Into  three  major  categories:  (a)  those  who  develop  and  modify 
logistics  capability  assessment  models  for  various  research  and  analysis  purposes,  (b)  those  who 
use  established  logistics  capability  asssessment  models  to  obtain  specified  outputs,  and  (c) 
those  Involved  with  technologies  similar  to  that  used  In  modeling  logistics  capability  assessment 


(for  exaaple,  coaputer-assisted  wargamlng).  The  Interviewees  Included  analysts  and  modelers  In 
the  following  organizations: 


Office  of  the  Joint  Chiefs  of  Staff,  Joint  Analysis  Directorate  (OJCS/JAD) 

HQ  US  Transportation  Coamand,  Analysis  and  Studies  Division  (TC/J5-E) 

HQ  US  European  Command,  Analysis  Division  (TC/J5-E) 

National  Defense  University,  Simulation  Wargaming  Center  (NDU/SWGC) 

HQ  USAF,  Readiness  Assessment  Group  (AF/XOOOC) 

Air  Force  Center  for  Studies  and  Analysis  (AFCSA): 

Tactical  Support  Division  (/SAGP) 

Fighter  Division  (/SAGF) 

HQ  Military  Airlift  Command  (HQ  MAC): 

Commander  In  Chief,  MAC  (CINCMAC)  Analysis  Group  (/AGR) 

Directorate  of  Manpower  and  Organization  (/XPM) 

War  Readiness  Material  (WRM)  Systems  Support  Section  (/LGSWR) 

HQ  Tactical  Air  Coimiand  (HQ  TAC): 

Tactical  Resources  Analysis  Office  (/LGY) 

Manpower  Studies  and  Analysis  Team  (/XPMS) 

Joint  Studies  Group  (/XPJSG) 

HQ  US  Air  Forces  In  Europe  (HQ  USAFE): 

Directorate  of  Operations  Analysis  (/DOA) 

Directorate  of  Manpower  and  Organization  (/XPM) 

Weapons  Systems  Support  Division  (/LGSW) 

Air  Force  Logistics  Management  Center:  Directorate  of  Logistics  Analysis  (AFLMC/LGY) 
Center  for  Aerospace  Doctrine  Research,  Development,  and  Evaluation  (AU  CADRE): 

Exercise  Operations  Olvislon  (/WGOO) 

Operations  Analysis  Division  (/WGTA) 

RAND  Corporation 


The  literature  review  was  very  useful  In  developing  a  broader  context  for  the  problems  of 
simulation  model  users.  The  references  discussed  the  original  purposes  behind  the  development  of 
the  models,  provided  a  history  of  model  development  and  uses,  and  demonstrated  that  some  of  the 
problems  associated  with  logistics  capability  assessment  models  are  coamion  to  other  types  of 
simulation  modeling.  Other  sources  were  useful  In  describing  general  theoretical  and  practical 
Issues  relative  to  developing  simulation  models.  Several  of  the  references  warrant  further 
discussion  here. 

An  excellent  reference  for  a  general  discussion  of  logistics  capability  assessment  Is  Nolte 
(1980).  It  describes  basic  concepts,  discusses  each  of  the  then-current  (several  are  still  In 
use)  capability  assessment  models,  and  provides  reconnendatlons  for  Improving  logistics 
capability  assessments. 

Rich,  Cohen,  and  Pyles  (1987)  trace  the  evolution  of  RAND's  research  on  the  resource 
management  Issues  Involved  In  assessing  combat  readiness  and  sustainability.  The  report 
discusses  some  of  RAND's  previous  modeling  efforts,  a  new  concept  known  as  Coupling  Logistics  to 
Operations  to  meet  Uncertainties  and  the  Threat  (CLOUT),  and  the  need  to  Incorporate  realistic 
assumptions  about  combat  operations.  In  particular  It  states: 

Current  analytic  techniques  do  not  adequately  reflect  the  likelihood  that  actual  flying 
patterns  may  differ  sharply  from  the  assumptions  In  our  planning  scenarios,  that  future 
component  removal  rates  (the  frequency  at  which  parts  need  maintenance  attention,  often 
relative  to  flying  hours  or  sorties)  will  probably  vary  (especially  during  wartime),  and 
that  airbase  damage  and  disruption  will  probably  shatter  our  assumptions  about  repair 
and  strck  resource  availability,  (p.  9) 
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Law  and  Kelton  (1982)  provide  a  comprehensive  discussion  of  simulation  modeling  and 
analysis.  Their  emphasis  Is  on  the  theoretical  statistical  Issues  Involved  with  probabilistic 
simulation  models. 

Bald  (1986)  discusses  current  problems  with  simulation  modeling.  The  problems  described  are 
very  similar  to  concerns  raised  In  the  Interviews;  for  example,  inadequate  documentation. 
Inadequate  management,  high  model  development  costs,  and  lack  of  full  life-cycle  support.  Bald 
provides  a  set  of  requirements  for  a  model  development  environment.  The  proposed  environment 
would  provide  an  Integrated  and  comprehensive  set  of  tools  for  cost-effective  and  automated 
support  of  a  model  throughout  Its  life  cycle. 


III.  USER- IDENTIFIED  SHORTFALLS  DF  CURRENT  MODELS 

This  section  sunnarlzes  the  data  collected  during  the  user  Interviews  and  describes  a  set  of 
"shortfalls"  Identified  by  various  users  of  capability  assessment  models.  All  shortfalls  cannot 
be  attributed  to  all  models;  however,  there  was  significant  connonallty  In  the  opinions  expressed 
by  users  of  different  models  as  well  as  by  different  organizations  using  the  same  model. 

The  users  surveyed  were  primarily  Involved  with  one  of  four  major  capability  assessment 
models  —  the  Theatre  Simulation  of  Airbase  Resources  (TSAR),  the  Dynamic  Multi-Echelon  Technique 
for  Recoverable  Item  Control  (Dyna-METRIC),  the  Logistics  Composite  Model  (LCOM),  and  the 
Tactical  Air  Coomand  (TAC)  Thunder  —  and  were  using  them  for  a  variety  of  management,  planning, 

and  analysis  purposes.  For  example,  the  WRM  Systems  Support  groups  at  all  conmands  were  using 

the  Weapon  System  Management  Information  System  -  System  Assessment  Module  (WS.'irS-SAM)  reporting 
capability,  based  on  the  Dyna-METRIC  model.  Dyna-METRIC  forecasts  demand  rate  per  flying  hour  of 
repairable  spare  parts;  however,  It  Is  currently  an  analytical  model  rather  than  a  true 

simulation.  The  manpower  comnunlty  Is  universally  using  LCOM  to  assess  maintenance  manpower 
Issues;  however,  LCOM  was  originally  designed  for  Investigating  nunerous  functional  areas  within 
logistics  capability  assessment.  The  Air  Staff  Is  using  a  general  purpose  model,  the  TSAR/TSAR 
Input  Using  Airbase  Damage  Assessment  (TSARINA)  to  perform  a  variety  of  special  studies  and  to 
perform  the  Air  Force  Capability  Assessment  Program  (AFCAP)  evaluation  of  unit  capability.  The 
Fighter  Division  In  the  AFCSA  Is  using  TAC  Thunder  for  a  large  group  of  operations  studies  and 
the  model  Is  also  being  developed  for  wargaming  exercises  at  various  locations.  RAND  Is 

continuing  research  and  development  on  new  versions  of  TSAR  and  Dyna-METRIC.  A  nunfcer  of  other 
models  were  also  encountered  that  were  not  used  frequently  enough  by  the  Air  Force  to  warrant 
Inclusion  In  this  study. 

The  Interview  results  were  organized  Into  a  set  of  basic  categories.  In  general,  the 
categories  below  describe  features  that  a  majority  of  users  felt  were  handled  poorly  by  their 
particular  model  or  by  Its  operating  environment. 


Data  Set  Repository 

Data  sets  do  not  belong  to  a  central  repository,  causing  redundancy  and  obsolescence. 
Although  large  amounts  of  resources  have  been  employed  In  creating  data  sets  for  various  models, 
there  Is  no  single  repository  where  these  expensive  data  sets  reside.  Rather,  each  organization 
that  creates  or  has  created  a  data  set  maintains  that  data  set  at  its  own  site. 


Data  Set  Currency 


Data  sets  are  often  not  current.  When  models  are  used  to  assess  the  combat  capability  of  Air 
Force  units,  the  data  sets  fed  Into  the  model  must  reflect  the  actual  state  of  the  organizational 
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unit  being  Modeled.  Automated  data  extraction  and  management  systems  could  alleviate  this 
problem.  Without  effective  data  base  management,  the  effort  required  In  maintaining  current  data 
Mill  remain  overly  demanding,  both  of  time  and  resources. 


Documentation 


Because  of  poorly  written.  Insufficient,  or  outdated  documentation,  almost  a11  the  model 
users  Interviewed  found  It  necessary  to  learn  to  use  the  model  from  the  original  developer  or  one 
of  his  disciples.  In  some  cases,  the  models  had  been  originally  developed  for  use  In  a  particular 
study  or  a  single  organization;  consequently,  the  developer  had  not  prepared  comprehensive 
documentation  for  the  model.  An  even  greater  problem  Identified  was  the  failure  to  provide 
adequate  documentation  updates.  The  documentation  for  these  often  large  models  becomes 
Increasingly  outdated  and  unreliable  as  the  models  evolve. 


Focusing  Capability 

Models  do  not  easily  permit  users  to  track  special  areas  of  Interest  throughout  the 
simulation.  This  requirement  might  Include  limiting  Input  or  output  to  only  the  areas  of 
concern,  or  tracking  a  given  variable  such  as  a  high  value  spare  or  a  particular  aircraft  tall 
number  throughout  the  simulation.  The  ability  to  focus  on  the  area  of  Interest  affects  not  only 
the  structure  of  the  models  themselves,  but  also  the  data  bases  upon  which  the  models  operate. 


Network  Development 


Model  activity  networks  are  difficult  to  change.  Each  of  the  four  major  capability 
assessment  models  makes  extensive  use  of  networks  to  establish  the  time  sequences  and 

relationships  among  simulated  activities.  In  these  models,  the  networks  are  exceedingly  complex 

but  subject  to  change  as  a  result  of  the  dynamic  nature  of  logistics  support  operations. 
Consequently,  the  operator  frequently  has  to  redefine  many  of  the  networks  which  constitute  the 

heart  of  the  model.  This  redefinition  currently  requires  labor-intensive  prograamlng  and  data 

preparation. 


OPFON  Operations 


Logistics  models  do  not  effectively  examine  the  Impact  of  Opposition  Force  (OPFOR)  operations 
on  *'e  logistics  support  posture  of  USAF  forces.  For  example,  an  OPFOR  employment  of  a 
"Fol  w-on-Forces  Attack"  that  concentrates  on  ensuring  no  aerial  resupply  by  strategic  airlift 
within  200  miles  of  the  Forward  Edge  of  the  Battle  Area  (FEBA)  would  have  a  significant  effect  on 
the  availability  of  logistics  support.  Current  models  only  allow  these  effects  to  be  modeled 
Implicitly  through  fixed  Input  parameters  (e.g.,  attrition  rates  and  supply  rates). 


Aircraft  Battle  Damage  Repair 


Most  model  users  felt  that  logistics  capability  assessment  models  do  not  treat  Aircraft 
Battle  Damage  Repair  (ABDR)  problems  effectively.  Frequently,  aircraft  attrition  simulations 
result  In  an  aircraft  which  does  not  return  to  base.  When  this  occurs,  the  models  assume  that 
none  of  the  damaged  aircraft  return  to  base.  As  a  result,  two  untoward  effects  occur.  First, 
those  aircraft  which  should  have  been  modeled  as  returning  to  base  damaged  do  not  enter  the 
maintenance  queue:  therefore,  the  maintenance  effort  which  would  be  devoted  to  returning  those 
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aircraft  to  serviceable  condition  Is  not  modeled.  In  addition,  the  maintenance  effort  required 
to  support  the  aircraft  once  returned  to  the  flying  schedule  Is  not  accounted  for.  Secondly,  the 
models  fall  to  consider  as  cannibalization  sources  those  battle-damaged  aircraft  which  are  In  a 
non-flyable  condition. 


Electronic  Coabat 

Existing  models  cannot  di'.tingulsh  between  missions  which  differ  as  to  their  requirements 
levied  upon  the  aircraft's  electronic  combat  equipment.  For  example,  not  one  of  the  models 
examined  provides  the  ability  to  tailor  EF-lllA  sorties  to  any  of  the  three  very  different  sortie 
profiles  which  might  be  demanded  and  which  place  enormously  different  stresses  upon  the 
aircraft's  mission  equipment.  Mission  profiles  that  vary  In  this  respect  would  have  differing 
equipment  failure  rates,  and  would  thus  place  differing  requirements  upon  maintenance  and  repair 
capabll  Itles. 


Electronic  Countermeasures  (ECM)  Equipment 


Spares  requirements  are  not  accurately  calculated  for  electronic  countermeasures  equipment. 
Both  TSAR  and  WSMIS-SAM  (Dyna-METRIC)  calculate  spares  requirements  based  upon  fallj'-e  rates 
loaded  Into  the  model.  In  today's  models  the  failure  history  of  the  components  Identified  In  the 
model  Is  based  upon  peacetime  failure  rates.  In  the  case  of  passive  ECM  equipment,  these  failure 
rates  are  probably  adequate  predictors  of  wartime  failure  rates.  For  active  ECM  equipment, 
however,  peacetime  failure  rates  (per  flying  hour)  do  not  approximate  wartime  use  sufficiently  to 
justify  their  use  for  wartime  scenarios. 


Operations  Models 


Current  models  do  not  accurately  reflect  rapidly  changing,  realistic  air  operations.  The 
operations  assumptions  built  Into  logistics  assessment  models  are  drivers  of  the  logistics 
workload;  however,  the  wartime  air  operations  which  the  operations  community  envisions  as  a 
future  possibility  are  not  built  Into  the  logistics  models.  Consequently,  the  logistics  problems 
associated  with  rapidly  changing  operational  scenarios  are  not  encountered.  Rapid  changes  of 
munitions  loads,  fuel  loads,  tactics,  electronic  demands,  or  eneay  tactics  are  all  areas  which 
cause  demands  on  logistics  to  fluctuate  wildly,  yet  they  are  not  explicitly  modeled  In  the 
logistics  models. 


Training 


No  systematic  training  on  model  construction,  modification,  or  use  exists.  In  that  most  of 
the  models  examined  are  large  and  use  specially  constructed  data  bases,  the  knowledge  required  to 
run  them  —  let  alone  modify  them  --  Is  extensive.  Further,  the  documentation  Is  often  weak  at 
best;  so,  new  operators  have  little  opportunity  for  self-paced  learning.  A  model  expert  or 
trainer  has  to  be  on  hand  virtually  all  the  time  if  new  people  are  to  have  an  opportunity  to 
learn  to  construct  and  run  the  models  effectively.  For  example,  the  LCOM  community  stated  almost 
unanimously  that  most  new  analysts  need  at  least  6  months  of  training  on  the  model  before  they 
can  be  considered  effective. 


Support  Operations 


Aircraft  missions  which  Include  support  operations  are  not  well  modeled.  For  Instance, 
neither  the  tanker  nor  the  receiver  aircraft  Is  well  modeled  for  air-to-air  refueling  efforts.  In 
addition,  the  effects  of  critical  component  failures  on  tanker  aircraft  are  not  accounted  for. 


5 


Mission  Packages 

All  three  of  the  mjor  logistics  capability  assessment  models  examined  assume  that  any 
aircraft  that  maintenance  can  deliver  can  be  used  Immediately  to  satisfy  unfulfilled  mission 
requirements.  This  Is  Incorrect.  Frequently,  operations  must  have  several  aircraft  available 
simultaneously  In  order  to  launch.  In  general,  tactical  aircraft  must  be  launched  In  pairs,  and 
mission  requirements  could  dictate  flights  of  four  to  twenty  as  the  minimum  required,  1n  addition 
to  any  support  aircraft. 


Attrition  Modeling 

Attrition  Is  currently  modeled  as  some  predetermined  loss  rate,  and  all  attritted  aircraft 
are  treated  as  "non-returners."  This  Is  an  unrealistic  approach  to  a  very  real  operational  and 
maintenance-related  problem.  As  discussed  under  ABDR,  returning  battle-damaged  aircraft  create 
maintenance  workloads,  yet  they  also  provide  a  source  of  cannibalized  parts. 


Pre-  and  Post-Processors 


The  models  examined  require  extensive  manipulation  of  data,  both  on  the  Input  and  the  output 
sides  of  the  model.  This  task  could  be  better  automated  through  statistics  processing  routines, 
graphics  Interface  programs,  or  graphics  display  packages.  This  requirement  Is  especially  severe 
with  the  LCOM  model,  where  each  major  manpower  study  requires  almost  a  complete  rewrite  of  the 
data  base. 


Level  of  Aggregatlon/firanularlty 

It  Is  often  the  case  that  the  level  of  detail  represented  In  the  model  or  Its  outputs  renders 
the  model  Inappropriate  or  cumbersome  for  use  In  a  particular  study*  (This  does  not  necessarily 
1iH)ly  that  the  model  will  not  be  used.)  The  presence  of  economies  or  diseconomies  of  scale 
Implies  that,  for  example,  a  wing-level  model  Is  not  simply  a  squadron-level  problem  Increased  by 
a  factor  of  three  or  four.  Thus,  the  ability  to  easily  select  the  level  of  aggregation  In  the 
model,  with  a  concomitant  adjustment  of  data  base  requirements,  would  be  highly  desirable.  This 
capability  Is  thwarted  In  current  models  by  a  lack  of  modularity,  coupled  with  fixed  data  base 
requirements  and  standard  output  products. 


Treatment  of  Partially  Mission  (^pable/Dual-Role  Aircraft 


Current  models  do  not  allow  the  analyst  to  examine  the  Increased  Operational  Plan 
supportabll Ity  which  can  be  obtained  from  proper  use  of  Partially  Mission  Capable  or  Dual-Role 
Aircraft.  If  an  aircraft  Is  Incapable  of  performing  even  one  of  many  possible  mission  types, 
today's  models  consider  It  Incapable  of  performing  any  mission.  Yet,  for  example,  an  F-16  which 
Is  not  capable  of  an  air-to-air  role  might  be  fully  capable  In  an  air-to-ground  role. 


Multiple  Base  Operations 

None  of  the  models  examined  has  the  ability  to  effectively  examine  the  capability  to  support 
air  operations  'rom  multiple  bases.  The  problems  associated  with  coamund,  control. 
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coHiunl cations,  lateral  resupply  of  cr1t<cal  parts,  and  reallocation  of  previously  allotted 
critical  parts  are  crucial  to  the  successful  conduct  of  wartime  operations  at  a  theater  level. 


Insti tuti onal 1 zatl on 


Soaie  users  were  very  concerned  by  the  lack  of  any  formal  Institutionalization  of  their 
model.  By  definition,  an  Institutionalized  model  Is  one  that  Is  widely  used  and  formally 
acknowledged  by  the  Air  Force  as  performing  certain  analysis  functions.  It  has  an  Office  of 
Primary  Responsibility  (OPR),  standardized  documentation  procedures,  training  classes,  standard 
Input  and  output  formats,  and  formalized  updating  procedures  (configuration  control). 


Configuration  Control 

This  Is  a  subset  of  the  Institutionalization  category,  and  refers  to  the  control  exercised  In 
maintaining  an  “official  version"  of  model  code.  Comparability  of  results  among  different  user 
organizations  can  be  assured  only  If  they  use  the  official  version  of  a  model.  Normally,  the  OPR 
would  establish  formal  procedures  for  Implementing  model  modifications  and  releasing  only  object 
code  versions  of  the  model. 


IV.  MODEL  EVALUATION  CRITERIA 

This  section  defines  a  set  of  criteria  for  the  evaluation  of  existing  or  proposed  capability 
assessment  models.  The  criteria  defined  are  based  on  Interviews  with  model  users  and  developers, 
and  study  of  the  literature  and  model  documentation  relevant  to  this  area. 

The  criteria  described  below  will  affect  the  cost  effectiveness  of  a  model,  the  mission 
effectiveness  of  a  model,  or  both.  Therefore,  we  may  think  of  these  two  measures  as  “global 
criteria"  that  provide  an  ultimate  measure  of  the  Importance  or  relevance  of  other  evaluation 
criteria  described  below.  Both  measures  are  first  discussed  to  provide  their  meaning  In  the 
context  of  logistics  capability  assessment. 


Cost  Effectiveness 


The  cost  effectiveness  of  a  model,  while  Important  to  consider  In  an  evaluation.  Is  difficult 
to  assess.  At  the  highest  level,  the  cost  effectiveness  of  a  model  Is  ultimately  derived  from 
the  value  of  the  Information  that  the  model  provides.  One  could  ask;  “Does  the  contribution  to 
defense  posture  provided  by  the  Information  justify  the  resources  necessary  to  provide  that 
Information?"  This  judgment  Is  necessari'y  subjective:  the  value  of  the  Information  Is  a 
function  of  not  only  the  model,  but  also  the  mission  of  the  using  organization,  the  people  using 
the  model,  the  accuracy  of  the  Input  data,  and  other  factors  often  beyond  the  control  of  the 
user.  A  measure  of  cost  effectiveness  that  Is  more  relevant  to  the  model's  using  organization 
could  be  given  by  determining  whether  the  Information  provided  by  a  model  could  be  provided  more 
cheaply  through  some  alternative  means.  Examples  of  these  alternatives  would  Include  empirical 
studies,  analytical  techniques,  expert  opinion,  or  other  simulation  models.  Under  this 
conceptual  framework,  cost  effectiveness  could  be  derived  primarily  from  comparisons  of 
cost-benefit  analyses  that  considered  development,  training,  data  preparation,  software 
maintenance,  and  hardware  acquisition  costs. 


7 


Mission  Effectiveness 


Considerations  of  Mission  effectiveness  provide  a  separate  measure  of  how  effective  the  model 
is  In  providing  Infonaatlon  that  contributes  to  the  mission  of  the  user's  particular 
organizational  unit,  Independent  of  the  model's  cost  effectiveness.  By  'mission,*  we  1iit>1y  a 
narrow  definition  that  refers  to  the  process  of  providing  specified  forms  of  Information 
(forecasts,  raw  data,  analyses,  etc.)  to  other  organizational  units.  Any  Improvement  to  a 
model's  Information-providing  capabilities  must  be  measured  against  Its  effects  on  mission 
effectiveness;  otherwise,  we  are  merely  modeling  for  modeling's  sake. 


Operating  Efficiency 

The  efficiency  of  a  model  can  be  evaluated  with  regard  to:  (a)  the  time  required  to  prepare, 
process,  and  analyze  Its  Input  and  output;  or  (b)  the  mode)  execution  time.  The  criterion 
described  here  refers  to  the  first.  The  total  time  Involved  here  Is  a  function  of  both 
model -dependent  features  and  features  that  arise  Independently  In  the  model's  operating 
environment.  An  example  of  a  model -dependent  feature  might  be  a  model  that  requires  the  user  to 
enter  the  data  In  a  very  precise.  Inflexible  format.  The  model  could  be  made  more  efficient  by 
removing  some  of  the  burden  placed  on  the  user,  thereby  reducing  time  required  for  data  entry. 
On  the  other  hand,  the  time  required  to  pre-process  Input  data  Is  often  a  function  of  factors 
that  are  Independent  of  the  model  but  still  affect  Its  operating  efficiency. 


Realism 


An  evaluation  of  simulation  models  should  Include  validation  of  the  simulation  results  as  an 
Important  factor  contributing  to  the  realism  and  credibility  of  the  models.  However,  It  Is 
extremely  difficult  and  costly  to  validate  military  simulation  models.  One  might  also  argue  that 
It  Is  Inappropriate  to  evaluate  a  simulation  model's  credibility  out  of  context;  that  Is,  apart 
from  Its  users.  Its  particular  application,  and  Its  Input  data.  Furthermore,  validation  Is  an 
ongoing  process.  In  that  the  systems  being  modeled  will  change  continually.  Practically  speaking 
then,  evaluation  of  a  model's  realism  by  knowledgeable  users  ('face  validity")  may  have  to 
suffice  as  validation.  Of  course  this  assumes  that  the  user  Is  sufficiently  knowledgeable  In 
those  functional  areas  to  render  such  a  Judgment. 

Models  are  constructed  to  provide  Information  In  specific  subject  domains,  at  given  levels  of 
detail,  and  under  assumptions  either  specified  or  Implied.  Judicious  exclusion  of  features 
perhaps  only  marginally  related  to  the  Issue  at  hand  Is  an  essential  part  of  sound  modeling 
practices.  Thus,  In  evaluating  realism  the  evaluator  must  do  so  only  In  terms  of  the  specific 
Information  desired  from  a  model.  For  example,  many  of  the  "shortfalls'  listed  In  the  previous 
section  of  this  paper  do  not  apply  to  every  situation;  for  particular  cases,  some  may  be 
Irrelevant.  The  following  list  provides  some  of  the  functional  areas  for  which  'realism'  within 
a  logistics  capability  assessment  model  could  be  evaluated: 


Multiple  base  operations 
Treatment  of  dual -role  aircraft 
ECM  and  Electronic  Combat 
OPFOR  Operations 
Aircraft  Battle  Damage  Repair 
Support  Operations 
Attrl tion 
Mission  Packages 


Maintenance  Networks 
Repairable  Spares 
Cannibalization  In  Peacetime 
Cannibalization  In  Wartime 
Tactics  (response  to  threat) 

Air  Base  Operability  (under  attack) 
Consumable  supplies 
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OocuMnt«t1on 


The  1^)ortance  of  good  docunentatlon  to  support  a  slnulatlon  nodel  cannot  be  overstated.  An 
Idea  has  value  only  If  the  Idea  can  be  comunlcated.  Slnllarly,  a  nodel  will  have  value  to  Its 
potential  users  only  If  they  can  understand  how  to  use  It.  If  the  documentation  does  not 
coMBunlcate  the  Infomatlon  that  the  user  needs,  the  nodel  nay  be  misused,  unused,  or  rejected  by 
the  user  In  favor  of  a  more  user-friendly  but  perhaps  less  appropriate  tool.  An  evaluation  of 
the  model  docunentatlon,  therefore.  Is  an  Important  part  of  model  evaluation.  It  should  Include 
an  evaluation  of  the  operational  system  docunentatlon,  the  support  system  documentation,  and  the 
documentation  control  plans  and  procedures. 


Data  Procedures 


The  effectiveness  of  logistics  capability  assessment  models  Is  contingent  upon  large 
quantities  of  rea1-wor1d  data  which  are  both  valid  and  consistent.  Many  classified  and 
unclassified  data  bases  are  maintained  by  the  Air  Force  for  planning  and  reporting  puiqposes. 
Simulation  developers  face  some  major  obstacles,  however,  when  they  attempt  to  use  these  data 
bases.  For  example,  they  often  receive  data  In  the  form  of  record-oriented  flat  files  which  In 
some  cases  must  be  subsequently  reorganized  Into  a  relational  data  base  and  loaded  Into  a  data 
base  management  system  (DBMS). 

Usually  the  external  data  bases  and  their  simulation-specific  counterparts  are  not  at  all 
Isomorphic.  For  one  thing,  generally  the  external  data  bases  generally  contain  much  more 
Information  than  Is  needed  for  a  particular  simulation.  The  user  must  therefore  extract  a  subset 
of  the  data  bases  In  preparing  the  Input  for  a  specific  simulation  model.  The  data  must  then  be 
transformed  again  to  meet  the  requirements  of  the  model,  and  then  be  reformatted  to  meet  the 
format  requirements  of  the  Input  data  sets. 

During  the  selection  of  the  appropriate  data  from  the  external  data  base  or  from  a  relational 
data  base,  the  semantics  of  the  data  play  a  major  role  In  the  Integration  and  abstraction 
operations  performed  by  the  model  user.  The  documentation  of  the  model  Is  critical  at  this  stage 
of  Input  preparation  because  In  the  course  of  Integrating  the  data  to  obtain  required  parameter 
values,  errors  can  be  made  that  are  very  difficult  to  detect  upon  review  of  the  Input  data  sets. 

Most  simulation  models  reviewed  In  this  study  were  found  to  be  executed  exclusively  In  batch 
mode.  As  an  alternative  to  batch  mode,  simulation  In  Interactive  mode  with  on-line  access  to 
external  data  bases  shows  promise  of  alleviating  the  problem  of  preparing  Input  data  sets.  A 
common  problem  with  reusing  data  files  Is  that  some  of  the  data  may  not  be  current  at  the  time 
the  file  Is  reused.  By  having  access  to  external  data  bases,  however,  the  data  can  be  frequently 

updated  and  saved.  Having  this  on-line  capability  can  greatly  reduce  the  burden  on  the  user  to 

update  the  Input  data  bases  and  can  also  Increase  the  credibility  of  the  simulation  by  ensuring 
the  currency  of  the  data. 

There  Is  no  single  criterion  for  evaluating  data  Input  procedures,  but  there  are  several 
related  criteria.  Clearly,  the  existence  of  an  established  data  set  repository  (discussed  In  the 
previous  section)  Is  directly  related,  and  would  alleviate  a  number  of  the  problems  discussed 

above.  The  value  of  a  data  repository  can  be  further  enhanced  through  residence  within  a  DBMS. 

One  of  the  features  of  a  DBMS,  however.  Is  that  It  must  be  upgraded  whenever  the  simulation  Is 
upgraded.  Its  usefulness  Is  greatly  reduced  If  It  Is  not  upgraded  concurrently  with  the  model, 
since  Its  value  lies  in  controlling  error  and  Inconsistency. 
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Power 


Discrete-event  siwlatlons  are  extrenely  limited  In  the  kinds  of  questions  they  can  answer. 
Typically  users  define  the  Initial  conditions,  run  the  simulation,  and  see  what  happens.  After 
obtaining  the  results,  they  ask  a  “what-lf  question,  redefine  the  Initial  conditions,  run  the 
simulation,  and  examine  the  output  again.  Running  the  simulation  and  drawing  a  conclusion  Is, 
In  effect,  drawing  an  Inference  from  an  explicit  representation  of  the  model.  Inference  can  be 
defined  In  general  terms  as  "the  ability  to  draw  Implicit  Information  from  explicit 
Information."  The  power  of  the  model  lies  In  the  ability  of  the  user  to  draw  an  Implicit 
conclusion  based  on  the  explicit  discrete-state  representation  of  the  problem. 

There  are  limitations,  however.  In  the  kinds  of  questions  that  discrete-state  simulations  can 
answer.  Although  these  simulations  can  answer  what-lf  questions,  there  are  other  kinds  of 
questions  that  are  equally  Important,  such  as  "Why?"  "When?"  "How?"  "Can  X  ever  be  Y?"  "What 
Is  the  highest/lowest  value  X  will  reach?"  or  "What  Initial  conditions  will  produce  the  maximum 
value  of  X?"  Because  of  the  what-lf  limitations,  the  conclusions  that  can  be  drawn  are  also 
generally  limited  to  statements  of  the  form  "Events  a,  b,  and  c  will  occur  under  Initial 
conditions  x,  y,  and  z." 

Consequently,  the  power  of  a  model  may  be  limited  by  Its  Inability  to  answer  various  types  of 
questions.  It  may  also  be  limited  In  Its  ability  to  represent  goals  and  Intentions,  plans  and 

beliefs.  Generally,  these  are  encoded  In  procedures,  making  It  difficult  to  answer  questions 

about  them  without  understanding  the  code.  As  a  measure  of  the  power  of  currently  used  models, 

users  can  rate  their  satisfaction  as  to  the  flexibility  the  model  provides  In  choosing  an 

aggregation  level,  and  Its  capability  to  focus  on  an  area  of  Interest. 


Output 


In  logistics  capability  assessment,  the  output  will  reveal  resource  utilization  rates  and  the 
number  of  sorties  generated.  In  the  case  of  wargaming  simulations,  the  output  may  Indicate 
relationships  between  players  at  given  points  In  time,  or  measures  of  effectiveness  such  as  miss 
distance  or  probability  of  kill.  Some  models  store  a  history  of  the  events  which  occur  In  the 
simulation,  providing  a  continuous  record  of  the  simulation.  Whereas  a  single  record  at  a  given 
point  In  time  Is  referred  to  as  a  snapshot,  a  continuous  record  Is,  In  effect,  a  data  base 
created  during  the  simulation  that  can  be  used  to  provide  Input  to  graphical  or  analytical 
routines.  These  routines  can  be  provided  by  a  post-processor  which,  like  the  simulation  model. 
Is  run  In  batch  mode.  In  other  cases,  the  graphics  processor  may  run  concurrently  with  the 
simulation  model  or  may  be  driven  by  the  simulation. 

Emerging  graphics  technologies  go  beyond  batch  mode  processing,  however,  to  allow  the  user  to 
explore  Interactively  not  only  what  occurred  In  the  simulation,  and  when  It  occurred,  but  also 
why  and  how  these  events  occurred.  Although  the  technologies  are  available,  the  degree  to  which 
they  have  been  Incorporated  Into  existing  simulation  models  varies  greatly  from  model  to  model. 


Output  Format 

A  simulation  model  can  be  evaluated  In  terms  of  how  well  Its  output  conveys  Information  to  a 
user.  Data  should  be  arranged  In  meaningful  patterns,  with  clear  descriptions  and  easy 
readability.  Dense  columns  of  numbers  with  cryptically  coded  header  descriptions  should  be 
avoided. 
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Graphics 


Graphics  can  be  applied  to  allow  greater  cotnprehensIbH Ity  of  simulation  Input,  output,  and 
state  during  the  model  execution.  Many  simulation  models  have  no  graphics  at  a11,  whereas  others 
have  some  graphical  representation  of  the  simulation  output.  A  more  advanced  use  of  graphics  1s 
found  In  the  development  of  the  Interactive  graphical  Interface. 

Graphical  Interfaces  to  simulations  make  possible  new  types  of  Interactions  by  allowing  one 
to  control,  manipulate,  and  monitor  simulations  of  dynamic  systems  at  many  different  hierarchical 
levels.  In  a  maintenance  support  system  simulation,  such  an  Interface  could,  for  example,  allow 
one  to  Inspect  the  status  of  a  key  component  as  1t  traverses  the  maintenance  network  from 
flightline  to  depot. 


Output  Analysis 

A  model  Is  of  little  value  unless  It  provides  output  which  Is  comprehensible  and  meaningful. 
Raw  output  data  will  usually  require  some  form  of  analysis  to  derive  a  meaningful  result.  In 
particular,  stochastic  models  necessitate  such  analyses  to  avoid  drawing  Improper  conclusions 
from  the  relatively  small  statistical  Information  available  In  the  output.  The  coR^juter  can  help 
alleviate  some  of  the  burden  of  performing  this  analysis.  For  example,  analytical 
post-processors  can  apply  a  series  of  standard  statistical  procedures  to  collections  of  output 
data.  Users  can  evaluate  their  modeling  environment  with  regard  to  Its  current  ability  to 
perform  and  facilitate  output  analysis. 


Design  Features 

The  design  of  a  model  is  Important  because  of  Its  direct  effects  upon  model 
comprehensibility,  ease  of  model  modification,  and  model  performance  and  efficiency.  A  properly 
maintained  model  is  not  static  but  Is  continually  modified  to  avoid  premature  obsolescence.  The 
architectural  design  Is  also  changing  as  a  result,  and  must  be  carefully  monitored  and  balanced 
to  ensure  that  a  good  overall  design  Is  maintained.  As  discussed  below,  this  task  may  not  be 
straightforward.  Tradeoffs  are  often  unavoidable.  For  example,  to  make  a  model  more  efficient, 
one  might  be  forced  to  reduce  Its  modularity.  However,  the  Increased  speed  of  available  computer 
hardware  now  allows  a  greater  emphasis  on  model  comprehensibility  over  efficient  source  code. 


Run-Time  Efficiency 


Architectural  features  that  affect  the  speed  of  execution  are  Important  to  note  In  an 
evaluation  of  a  model  because  of  the  strong  relationship  between  run-time  efficiency  and  total 
costs.  Data  structures  should  be  designed  for  efficient  data  handling  during  model  execution. 
Previous  software  development  techniques  emphasized  control  structures  as  opposed  to  the 
organization  of  the  data  structures  (Booch,  1983).  Less  attention  was  paid  to  the  handling  of 
global  variables  and  module  Interface  design  (Dickinson  t  Steenrod,  1984).  Good  architectural 
design  requires  a  balanced  treatment  of  control  structures  and  data  structures.  Criteria  for 
model  evaluation  should  therefore  Include  criteria  for  well-designed  data  structures,  good 
handling  of  shared  data,  efficient  control  structures,  and  wel 1 -designed  module  Interfaces. 

Modularity 

Modularity  is  of  particular  Importance  because  of  Its  effects  on  the  comprehensibility  and 
flexibility  of  the  source  code.  This  principle  can  be  applied  In  different  ways  according  to 
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different  design  wthodol ogles,  resulting  In  differing  degrees  of  flexibility  In  design.  For 
exanple,  top-down  structured  design,  as  described  by  Yourdon  and  Constantine  (1979),  suggests 
that  a  systea  be  decoaposed  by  aaklng  each  step  In  the  process  a  module.  This  leads  to  aodules 
that  are  highly  functional  and  focuses  on  operations  rather  than  data,  leaving  the  data  structure 
to  be  defined  later.  This  concept  Is  best  exemplified  In  object-oriented  progrannlng  languages. 
At  the  other  end  of  the  spectrum,  there  are  alternative  design  methodologies  that  define  data 
structures  first,  and  then  structure  the  program  units  based  on  the  data  structure.  Using  this 
technique,  one  can  clearly  define  the  Implementation  of  the  objects  In  the  solution  space,  and 
then  make  their  structure  visible  to  the  necessary  functional  units  that  provide  the  operations 
of  the  objects. 

A  simulation  model  can  be  evaluated  for  Its  degree  of  modularity  by  examining  the  length  of 
the  subroutines,  and  the  design  of  the  module  Interfaces.  Long  subroutines  Imply  a  need  to 
further  decompose  the  code  Into  smaller  modules.  The  Interfaces  should  have  a  standard  format, 
and  their  preambles  should  be  accurate  and  consistent  with  the  source  code. 


Ease  of  Ptodifl cation 


Models  may  require  modification  for  one  of  three  reasons:  (a)  to  respond  to  a  change  In 
requirements  for  the  modeled  system,  (b)  to  add  enhanced  modeling  capabilities  or  levels  of 
detail,  or  (c)  to  correct  errors  that  have  been  Introduced  earlier  In  the  development  process.  A 
model  that  Is  modular  In  Its  design  Is  more  easily  modified  because  changes  can  be  Introduced 
without  Increasing  the  complexity  of  the  system.  The  effects  of  changes  tend  to  be  localized 
within  the  module  where  the  change  occurs.  If  the  module  Is  small  In  size.  It  Is  easier  to 
predict  where  these  effects  will  occur.  Hence,  It  Is  easier  to  control  aqy  undesirable  effects 
of  Introducing  a  change. 

Enhanceaients  of  the  software  must  be  designed  within  the  existing  structure  of  the  model. 
Because  earlier  languages  such  as  Fortran  are  not  self-documenting,  the  programmer  must  rely  on 
external  documentation  to  represent  the  structure.  Capability  assessment  models  have  tended  to 
be  written  In  either  Fortran  or  Simscript;  thus,  an  evaluation  of  the  documentation  Is  Important 
to  understand  how  well  the  structure  Is  conveyed  and  to  estimate  the  labor  that  would  be  required 
to  enhance  the  model  or  to  correct  errors.  If  the  structure  Is  not  modular  and  the  documentation 
Is  Incomplete,  a  major  upgrade  In  the  existing  model  may  be  required  before  ary  real  enhancements 
can  be  designed  by  anyone  other  than  the  original  model  developers. 


Size  of  the  Program 

Models  can  be  evaluated  In  terms  of  the  number  of  executable  statements.  The  size  of  a 
program  Is  one  factor  In  determining  the  difficulty  of  upgrading  or  modifying  the  program; 
however,  this  problem  can  be  reduced  with  Increased  modularity  of  design.  On  the  other  hand, 
the  ability  to  verify,  validate,  or  fully  comprehend  a  model  will  certainly  decrease  as  a 
function  of  Increased  model  size. 


ial  ementatlon  Language 

The  choice  of  language  will  affect  the  degree  of  modularity,  the  number  of  lines  of  source 
code,  the  comprehensibility  of  the  code,  and  training  and  maintenance  costs.  An  organization 
must  be  able  to  locate  analysts  who  are  fluent  In  the  language  of  the  model,  or  provide 
training.  Initial  learning  or  retraining  can  be  costly.  Simulation  models  may  be  written  In 
general  purpose  progrannlng  languages  such  as  Fortran,  Pascal,  Basic,  or  Ada.  The  advantages  of 


12 


using  one  of  these  languages  are  the  relatively  large  nianber  of  analysts  who  know  tnese 
languages,  and  the  easy  portability  between  computer  systems  due  to  the  fact  that  compilers  for 
these  languages  are  fairly  common.  Specialized  simulation  and  modeling  languages  also  exist 
(SLAM,  GASP,  GPSS,  Simscript).  Fluency  In  these  languages  Is  less  common;  however,  model 
development  time  can  be  significantly  reduced  In  that  each  statement  In  a  higher  level  simulation 
language  can  replace  numerous  statements  In  a  general  purpose  programnlng  language.  In  fact, 
most  simulation  languages  are  constructed  from  a  general  purpose  language. 


Treatment  of  Randomness 


Simulation  models  often  assume  that  physical  functions  can  be  modeled  as  stochastic 
processes.  This  allows  the  Incorporation  of  elements  of  natural  randomness.  On  the  other  hand. 
It  entails  building  Implicit  assumptions  Into  the  model  regarding  the  form  and  parameters  of 
numerous  probability  distributions.  Verifying  the  statistical  validity  of  these  distributions 
and  maintaining  this  validity  over  time  are  arduous  tasks.  At  the  same  time,  seme  types  of 
desired  model  outputs  may  be  rather  Insensitive  to  the  accuracy  of  Internal  probability 
distributions.  In  such  cases,  an  expected  value  treatment  may  be  more  appropriate.  This  Is 
especially  true  If  only  a  rough  first-order  type  of  analysis  Is  needed  for  a  given  system. 


Institutionalization 

Institutionalization  of  a  model  carries  with  It  two  very  Important  attributes.  First  and 
foremost  Is  the  recognition  that  a  particular  model  Is  the  preferred  vehicle  for  performing  a 
particular  task.  From  this  recognition  flows  widespread  use  of  the  model  and  acceptance  of  model 
results  as  being  authoritative  representations  of  the  modeled  scenario.  Secondly,  Institu¬ 
tionalization  requires  that  some  agency  be  designated  as  Office  of  Primary  Responsibility  (OPR) 
for  the  model.  From  this  OPR  designation  stems  the  responsibility  for  configuration  management 
of  the  model  Including  documentation,  currency,  and  the  other  activities  that  responsibility 
entails. 


Configuration  Management 

Configuration  management  Is  the  application  of  technical  and  adninistrative  direction  (a)  to 
Identify  the  configuration  items  that  comprise  the  model  baseline;  (b)  to  control  the  changes 
made  to  the  baseline;  and  (c)  to  record  and  report  change  processing  and  Implementation  status. 
The  technical  and  administrative  direction  Is  carried  out  according  to  a  Configuration  Management 
Plan,  which  specifies  controls  that  are  required  for  maintainability  and  provides  a  system  for 
recording  errors  Identified,  errors  corrected,  and  enhancements  made.  Upgrades  made  by  users  In 

different  locations  need  to  be  coordinated  and  controlled  to  conserve  resources.  This 

coordination  necessitates  the  establishment  of  a  baseline  for  the  model  and  a  plan  for 

maintaining  configuration  control.  Configuration  Management  may  also  have  the  effect  of  opening 
up  to  a  larger  number  of  contractors  the  opportunity  to  work  on  the  model,  encouraging 
competition  for  the  opportunity  to  participate.  Involving  a  larger  number  of  contractors  makes 
the  need  for  the  standards  more  critical  and  more  obvious  to  model  developers. 

Criteria  for  model  evaluation  should  therefore  Include  an  examination  of  the  way  In  which  the 
model  Is  maintained,  whether  It  Is  under  configuration  control,  whether  Its  modifications  are 
designed  to  meet  existing  software  standards,  and  whether  It  has  a  User's  Group  and  other 

organized  support  groups. 
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Y.  EVALUATION  METHODOLOGY 


This  section  Illustrates  how  to  use  the  criteria  In  a  model  evaluation.  The  purpose  of  the 
evaluation  methodology  described  Is  to  provide  using  organizations  with  a  means  of  making  an 
Intelligent  choice  between  two  or  more  candidate  models  or  modeling  systems,  either  proposed  or 
already  In  existence. 

This  evaluation  Is  Inherently  subjective.  There  Is  no  absolute  scale  for  measuring,  say,  the 

quality  of  the  output  format  or  the  efficiency  of  the  data  procedures.  Furthermore,  we  must 

establish  some  unified  basis  for  making  tradeoffs  among  different  and  difficul t-to-quantify 
attributes.  Rather  than  relying  on  Intuition  alone,  however,  we  do  have  some  recourse  In 
quantitative  techniques  having  firm  theoretical  foundations.  He  refer  specifically  to  the 
techniques  of  utility  or  preference  functions  (Keeney  A  Ralffa,  1976). 

The  strong  point  of  utility  theory  Is  Its  reliance  on  the  preferences  of  the  decision-makers 
themselves  In  assessing  values  for  the  components  of  an  evaluation.  If  we  assume  that  the 

decision-maker  will  make  "rational''  assessments,  the  Identified  course  of  action  will  be 

"optimal*  for  the  decision-maker.  This  rationality  will  depend  on  the  consistency  of  the 
decision-maker's  stated  preferences. 


Problem  Structure 


We  shall  structure  our  problem  such  that  a  utility  Is  to  be  determined  for  each  candidate 
-model /system.  The  model/system  with  the  highest  overall  utility  Is  then  the  optimal  choice  from 
the  decision-maker's  point  of  view.  (The  utility  of  each  model  Is  to  be  calculated  as  a  weighted 
function  of  the  evaluation  criteria  and  the  estimated  life-cycle  costs  of  using  the  model/ 
system.)  To  simplify  the  discussion,  we  will  assume  that  the  decisions  are  to  be  made  under 
certainty:  that  Is,  that  outcomes  are  known  In  advance. 

He  propose  that  each  model /system  under  consideration  be  provided  with  two  relevant 
attributes  that  must  be  determined:  mission  effectiveness  and  cost  (negatively  oriented).  These 
attributes  are  derived  from  the  two  "global  criteria,"  cost  effectiveness  and  mission 
effectiveness,  discussed  In  the  previous  section.  Mission  effectiveness  was  defined  as  a  measure 
of  how  well  a  model  assists  a  user  organization  In  Its  mission  of  providing  specified  forms  of 
Information  (forecasts,  raw  data,  analyses,  etc.)  to  other  Air  Force  organizations.  The  cost 
attribute  refers  to  the  organization's  estimated  life-cycle  costs  of  using  a  particular 
model /system.  Reducing  the  problem  to  these  two  attributes  focuses  the  analysis  on  their 
Inevitable  tradeoff. 

The  mission  effectiveness  of  a  model /system  will  be  constructed  as  a  weighted  linear 
combination  of  Its  mission  effectiveness  when  measured  against  each  Individual  criterion.  For 
clarity  In  this  discussion,  we  define  the  following  parameters: 

K^  -  the  estimated  life-cycle  cost  for  model  1, 

M^  -  the  overall  mission  effectiveness  of  model  1, 

r^j  -  the  mission  rating  of  model  1  when  measured  against  criterion  j, 

Wj  -  the  relative  Importance  of  criterion  j  In  relation  to  the  mission  of  the  using 
organization. 

The  relationship  between  mission  effectiveness  parameters  Is  defined  here  as: 

*^1  "  ^  j  *'j'’1j  n) 
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The  range  of  allowable  values  for  the  parameters  can  be  set  arbitrarily,  as  long  as  It  Is 
consistent  across  the  criteria  and  between  models.  Here  we  assign  r^j  to  the  Interval  li 
''ij  i  5  and  Wj  to  the  Interval  0  ^  wj  ^  1.  In  the  case  of  r^j,  a  value  of  1  would 

Indicate  that  model  1  has  poor  performance  for  criterion  J,  whereas  a  value  of  5  would  Indicate 
superior  performance.  In  the  case  of  wj,  a  value  of  0  would  Indicate  that  criterion  j  is 
Irrelevant  to  the  mission  of  the  using  organization;  a  value  of  1  would  Indicate  that  the  model's 
performance  for  criterion  J  must  be  maximized. 

The  other  attribute  of  the  utility  function  Is  total  life-cycle  cost.  Estimation  of  this 
cost  should  be  done  as  accurately  as  possible,  acknowledging  that  some  costs  can  be  only  roughly 
approximated.  A  complete  discussion  of  costing  techniques  Is  beyond  the  scope  of  this  paper. 
For  further  Information  see,  for  example.  Grant,  Ireson,  and  Leavenworth  (1982),  Air  Force 

Systems  Connand  (19851,  or  Electronic  Systems  Division  (1984). 

Recall  that  Is  the  estimated  life-cycle  cost  for  model  1.  Our  utility  function  can  then 
be  expressed  as: 

U^  *  u(M^,  K^)  (2) 

where  u  Is  an  Increasing  function  of  the  Independent  variables  and  K^.  The  precise  form 

of  the  function  will  depend  on  the  preferences  expressed  by  the  decision-maker  regarding  the 
rates  of  substitution  between  the  two  variables.  For  example,  under  certain  conditions  the 

function  u  may  be  additive,  such  that  «  m(M^)  *  k(K{),  where  m  and  k  are  both  Increasing 
functions. 

Establishing  the  form  of  the  function  u  Is  accomplished  by  having  the  decision-maker  answer 
questions  of  the  following  form:  If  Is  Increased  by  some  number  of  units,  by  how  many  units 
does  K.{  have  to  decrease  (costs  Increase)  for  you  to  be  Indifferent?  (That  Is,  at  what  point 
would  you  feel  that  Increased  costs  make  a  given  increase  In  mission  effectiveness  of  neutral 
value.)  In  most  cases,  the  answer  depends  on  the  current  levels  of  Nj  and  K^.  If  at  level 
(M,  K)  of  and  the  decision-maker  Is  willing  to  give  up  pi  units  of  for  ,  then 
the  "marginal  rate  of  substitution*  of  for  M.(  at  (M,  K)  Is  p.  The  marginal  rate  of 
substitution  Is  defined  as  the  negative  reciprocal  of  the  slope  of  the  "Indifference  curve*  at 
(M,  K).  (See  Figure  1.) 


Figure  1.  Indifference  Curves. 
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The  decislonniHiker  Is  Indifferent  between  any  combination  of  M.|  and  on  a  given 
Indifference  curve.  Moving  outward  from  the  origin,  the  curves  represent  higher  levels  of 
satisfaction  (utility)  for  the  decision-maker.  The  curves  are  constructed  by  plotting  data 
corresponding  to  the  decision-maker's  marginal  rates  of  substitution.  (See  Keeney  and  Ralffa  for 
examples  and  further  discussion  of  how  one  might  structure  an  Interview  to  obtain  these  data.) 
For  each  model  1,  the  points  M^  and  K^  are  plotted  on  the  same  graph.  The  model 
corresponding  to  the  point  on  the  highest  Indifference  curve  is  selected  as  optimal  for  the 
decision-maker. 


Example 


Table  1  presents  a  list  of  model  evaluation  criteria  and  sample  mission  Importance  ratings 
(wj  values)  assigned  by  a  hypothetical  decision-maker.  The  criteria  are  divided  Into  two 
categories:  "general  criteria"  and  "domain  criteria."  General  criteria  apply  to  most  large 
discrete-event  simulation  models.  The  domain  criteria  are  those  that  might  be  applied  to  models 
In  the  functional  area  of  logistics  capability  assessment.  The  relevance  of  each  of  the  criteria 
to  our  hypothetical  decision-maker  Is  reflected  by  Its  Wj  value. 

Table  1.  Sample  Mission  Importance  Ratings 


General  criteria 

Domain  criteria 

Power 

0.83 

Multiple  Base  Operations 

0.92 

Output  Analysis 

0.83 

Attrition 

0.92 

Ease  of  Modification 

0.75 

Mission  Packages 

0.92 

Operating  Efficiency 

0.75 

Aircraft  Battle  Damage  Repair 

0.83 

Documentation 

0.75 

Repairable  Spares 

0.83 

Data  Procedures 

0.75 

Airbase  Operability 

0.83 

Modularity 

0.75 

Consumables 

0.75 

Configuration  Management 

0.75 

Cannibalization  -  Wartime 

0.75 

Run-Time  Efficiency 

0.75 

Dual -Role  Aircraft 

0.75 

Output  Format 

0.67 

ECM  and  Electronic  Combat 

0.67 

Treatment  of  Randomness 

0.58 

Support  Dperatlons 

0.67 

Implementation  Language 

0.42 

OPFOR  Operations 

0.56 

Output  Graphics 

0.42 

Maintenance  and  C2  Networks 

0. 50 

Size  of  Program 

0.17 

Cannibalization  -  Peacetime 

0.13 

Tactics 

0.10 

Table  2  provides  model  criteria  ratings  (r^j  values)  for  three  hypothetical  candidate 
models/systems,  denoted  here  as  A,  B,  and  C.  When  the  criteria  ratings  are  multiplied  by  the 
mission  Importance  ratings  from  Table  1,  and  sunned  for  each  model,  we  arrive  at  the  mission 
effectiveness  attributes  listed  at  the  bottom  of  Table  2. 
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Table  2.  Model  Criteria  Ratings 


Criteria  ratings  (r^j) 


Criterion 

Model  A 

Model  B 

Model  C 

(Wj) 

Data  Procedures 

1.33 

3.33 

3.00 

0.75 

ECM  and  Electronic  Combat 

2.00 

3.00 

4.00 

0.67 

Configuration  Management 

1.67 

4.17 

2.25 

0.75 

Output  Analysis 

2.33 

3.54 

2.75 

0.83 

Output  Graphics 

1.33 

4.00 

1.00 

0.42 

Ease  of  Modification 

1.25 

4.33 

3.00 

0.75 

Output  Format 

3.00 

4.00 

3.00 

0.67 

Modularity 

1.50 

4.33 

4.00 

0.75 

Cannibalization  -  Peacetime 

3.67 

4.33 

4.00 

0.13 

Size  of  Program 

1.33 

4.50 

1.50 

0.17 

Documentation 

2.50 

3.50 

2.75 

0.75 

Operating  Efficiency 

2.00 

5.00 

3.75 

0.75 

Tactics 

1.00 

1.33 

1.00 

0.10 

Attrition 

3.00 

3.33 

4.50 

0.92 

Aircraft  Battle  Damage  Repair 

3.00 

1.00 

3.50 

0.83 

Mission  Packages 

3.33 

2.67 

1.00 

0.92 

Cannibalization  -  Wartime 

3.67 

4.33 

4.00 

0.75 

Dual -Role  Aircraft 

4.00 

2.00 

3.00 

0.75 

Treatment  of  Randomness 

4.00 

4.67 

3.50 

0. 58 

OPFOR  Operations 

3.00 

1.00 

1.00 

0.58 

Implementation  Language 

3.33 

4.50 

1.25 

0.42 

Maintenance  and  C2  Networks 

4.67 

2.33 

5.00 

0.50 

Consumables 

4.00 

3.33 

4.00 

0.75 

Power 

3.50 

4.17 

4.12 

0.83 

Multiple  Base  Operations 

3.00 

3.00 

3.25 

0.92 

Run-Time  Efficiency 

4.00 

5.00 

3.75 

0.75 

Support  Operations 

5.00 

3.00 

4.75 

0.67 

Airbase  Operability 

4.33 

1.33 

2.00 

0.83 

Repairable  Spares 

4.67 

5.00 

5.00 

0.83 

Mission  Effectiveness  Ratings: 

M/^  -  58.58 

Mb  -  66.44 

Me  -  62.39 

(  Mf 

) 

Based  on  the  aiission  effectiveness  ratings  alone,  model /system  B  Is  the  best,  followed  by  C 
and  then  A.  Before  making  a  selection,  however,  life-cycle  costs  need  to  be  considered.  Assune 
that  the  following  costs  have  been  calculated; 

Model /System  A  -  $2,OOOK 
Model /System  B  -  J3,OOOK 
Model /System  C  -  $2,500K 

The  (M^,  k^)  points  can  now  be  plotted.  (See  Figure  2  -  Note  that  the  cost  axis  Is 
negatively  oriented.)  Clearly,  the  optimal  choice  will  depend  on  the  form  of  the  indifference 
curves,  since  no  point  "dominates"  the  others.  That  Is,  no  model/system  Is  both  highest  In 
mission  effectiveness  and  lowest  In  cost. 
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Figure  2.  Results  of  ExMple. 

In  this  exaaple,  the  Indifference  curves  have  been  plotted  such  that  model  C  achieves  highest 
overall  utility.  This  occurred  despite  the  fact  that  It  was  neither  the  most  mission  effective’ 
nor  lowest  In  cost.  Different  preferences  by  the  decision-maker  on  the  rates  of  substitution 
between  and  could  have  produced  other  results,  however. 


Limitations 


There  are  possible  limitations  to  the  literal  application  of  the  evaluation  methodology. 
First,  In  an  attempt  to  be  comprehensive,  the  method  focuses  attention  on  distinct  categories  of 
model  evaluation  criteria.  The  astute  reader  has  probably  recognized  that  these  categories  are 
nelthe'’  necessarily  distinct  nor  Independent;  In  fact,  some  are  highly  related.  This  may 
Introduce  an  element  of  bias  Into  the  evaluation  In  that  model  deficiencies  may  get 
"double-counting"  If  they  affect  more  than  one  criterion.  For  example,  a  very  large  program 
could  receive  a  poor  rating  In  "size  of  program."  It  might  also  get  poor  ratings  In  "run-time 
efficiency"  or  "ease  of  modification,"  simply  because  it  Is  a  large  program.  On  the  other  hand, 
large  programs  are  not  necessarily  Inefficient  or  difficult  to  modify;  thus,  the  problem  may  not 
be  significant. 

Second,  It  may  be  difficult  for  a  decision-maker  to  determine  substitution  rates  between 
dollar  costs  and  an  arbitrarily  scaled  mission  effectiveness  rating.  This  problem  can  be 
ameliorated  by  allowing  the  decision-maker  to  first  develop  an  Intuitive  notion  of  the  value  of 
"a  unit*  of  mission  effectiveness.  In  addition.  It  may  be  that  critical  values  can  be 
established  for  the  marginal  rate  of  substitution  such  that,  for  example,  M  ^  M*  Implies 

model/system  A  Is  optimal ,  4'^  U-  ^  u- ^  Implies  model/system  B  Is  optimal,  etc.  In  this 

way.  the  decision-maker  need  not  determine  precise  values  for  substitution  rates,  but  need 

determine  only  within  which  of  several  broad  ranges  the  values  fall. 

It  seems  clear  that  sensitivity  analysis  can  play  a  major  role  In  the  model  evaluation.  This 

Is  true  not  only  In  the  case  of  marginal  rates  of  substitution  above,  but  also  In  assigning 

values  to  the  model  criteria  and  relative  mission  Importance  ratings.  Sensitivity  analysis  may 
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also  provide  relief  to  those  attempting  precise  calculations  of  life-cycle  costs.  A  general  rule 
of  *humb  Is  to  first  determine  the  sensitivity  of  the  solution  to  small  changes  In  parameter 
values  before  expending  a  great  deal  of  effort  In  deriving  precise  values  for  the  parameters. 


VI.  CONCLUSIONS 

This  paper  Illustrated  that  evaluating  any  simulation  model  requires  consideration  of 

numerous  criteria.  However,  not  all  criteria  are  equally  relevant  In  all  situations  or  to  all 
organizations.  The  proposed  evaluation  methodology  takes  this  fact  Into  account  through  the  use 
of  mission  1mporta.ce  ratings.  These  ratings  also  provide  a  convenient  vehicle  for  the 

expression  of  a  decision-maker's  preferences,  a  necessary  element  In  deriving  utility  functions. 

A  corollary  to  the  notion  of  evaluation  criteria  having  differing  degrees  of  relevance  or 
Importance  among  organizations  Is  the  fact  that  the  Importance  ratings  will  change  over  time 

within  a  single  organization.  Such  changes  stem  from  the  highly  dynamic  nature  of  Air  Force 
operations  and  procedures,  as  well  as  the  availability  of  new  modeling  technologies.  Ultimately, 
any  model  will  eventually  become  obsolete.  Organizations  must  therefore  continually  monitor  and 
evaluate  models  to  ensure  that  the  models'  performance  (mission  effectiveness)  and  cost 
effectiveness  justify  continued  usage.  It  Is  hoped  that  this  paper  will  provide  a  basis  for  such 
evaluations. 

We  have  also  Illustrated  the  Inevitable  tradeoff  between  model  costs  and  model  performance 
that  must  be  faced  by  decision-makers.  Not  as  apparent,  however,  are  the  tradeoffs  among 
performance  criteria.  However,  such  tradeoffs  must  be  made,  as  ho  model  can  be  all  things  to  all 
people.  For  example,  attempts  to  continually  Incorporate  additional  "realism"  and  detail  Into  a 
model  will  eventually  make  the  model  too  cumbersome  and  difficult  to  use.  In  terms  of  the 

criteria  given  here,  run-time  efficiency,  operating  efficiency,  and  ease  of  modification  would 
eventually  deteriorate  In  the  face  of  repeated  program  modifications.  A  more  effective  strategy 
would  be  to  develop  models  that  would  provide  for  a  high  degree  of  flexibility.  Thus,  models 
could  be  altered  or  extended  without  necessarily  making  them  mere  complicated  or  less  efficient. 
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